[core] refactor: extract and centralize HTTP fetcher by KristjanESPERANTO · Pull Request #4016 · MagicMirrorOrg/MagicMirror

KristjanESPERANTO · 2026-01-20T00:36:34Z

Summary

PR #3976 introduced smart HTTP error handling for the Calendar module. This PR extracts that HTTP logic into a central HTTPFetcher class.

Calendar is the first module to use it. Follow-up PRs would migrate Newsfeed and maybe even Weather.

Before this change:

❌ Each module had to implemented its own fetch() calls
❌ No centralized retry logic or backoff strategies
❌ No timeout handling for hanging requests
❌ Error detection relied on fragile string parsing

What this PR adds:

✅ Unified HTTPFetcher class with intelligent retry strategies
✅ Modern AbortController with configurable timeout (default 30s)
✅ Proper undici Agent for self-signed certificates
✅ Structured error objects with translation keys
✅ Calendar module migrated as first consumer
✅ Comprehensive unit tests with msw (Mock Service Worker)

Architecture

Before - Decentralized HTTP handling:

Calendar Module          Newsfeed Module         Weather Module
┌─────────────┐         ┌─────────────┐         ┌─────────────┐
│ fetch() own │         │ fetch() own │         │ fetch() own │
│ retry logic │         │ basic error │         │ no retry    │
│ error parse │         │   handling  │         │ client-side │
└─────────────┘         └─────────────┘         └─────────────┘
      │                       │                       │
      └───────────────────────┴───────────────────────┘
                              ▼
                        External APIs

After - Centralized with HTTPFetcher:

┌─────────────────────────────────────────────────────┐
│                  HTTPFetcher                        │
│  • Unified retry strategies (401/403, 429, 5xx)     │
│  • AbortController timeout (30s)                    │
│  • Structured errors with translation keys          │
│  • undici Agent for self-signed certs               │
└────────────┬──────────────┬──────────────┬──────────┘
             │              │              │
     ┌───────▼───────┐ ┌────▼─────┐ ┌──────▼──────┐
     │   Calendar    │ │ Newsfeed │ │   Weather   │
     │   ✅ This PR  │ │  future  │ │   future    │
     └───────────────┘ └──────────┘ └─────────────┘
             │              │              │
             └──────────────┴──────────────┘
                          ▼
                   External APIs

Complexity Considerations

Does HTTPFetcher add complexity?

Even if it may look more complex, it actually reduces overall complexity:

Calendar already has this logic (PR [calendar] fix: prevent excessive fetching on client reload and refactor calendarfetcherutils.js #3976) - we're extracting, not adding
Alternative is worse: Each module implementing own logic = 3× the code
Better testability: 443 lines of tests once vs. duplicating tests for each module
Standards-based: Retry-After is RFC 7231, not custom logic

Future Benefits

Weather migration (future PR):

Moving Weather from client-side to server-side will enable:

Same robust error handling - Weather gets 429 rate-limiting, 5xx backoff for free
Simpler architecture - No proxy layer needed

Moving the weather modules from client-side to server-side will be a big undertaking, but I think it's a good strategy. Even if we only move the calendar and newsfeed to the new HTTP fetcher and leave the weather as it is, this PR still makes sense, I think.

Breaking Changes

None

I am eager to hear your opinion on this 🙂

- Extract HTTP logic from Calendar module into HTTPFetcher class - Migrate calendar module to use centralized HTTPFetcher - Add modern improvements: * AbortController with configurable timeout (default 30s) * undici Agent for self-signed certificates * Improved Retry-After header handling (respects server hints) - Implement intelligent retry strategies: * 401/403: 5x interval, min 30 minutes * 429: Retry-After header parsing, fallback 15 minutes * 5xx: Exponential backoff (max 3 retries) * 4xx: 2x interval, min 15 minutes - Provide structured error objects with translation keys - Comprehensive unit tests with msw (Mock Service Worker) - Tests cover all error scenarios and authentication methods

rejas

Awesome idea, I really like!

Just one thing: We dont add the default english translations to other languages. If a translation isnt available, it automatically takes the english one.

KristjanESPERANTO · 2026-01-21T10:13:16Z

My idea was to make it easier for the translators, but you're right, that was not very clean. Instead of taking the easy way out and simply removing them, I used automatic translation (which has delivered very good results in the past) to translate the strings. In doing so, I even noticed duplicate strings in the pt translation, which I removed right away.

translations/pt.json

khassel · 2026-01-22T22:32:26Z

This broke my calendar:

[2026-01-22 23:27:21.769] [ERROR] [calendar] https://calendar.google.com/calendar/ical/xxx.calendar.google.com/public/basic.ics - iCal parsing failed: UNTIL rule part MUST have the same value type as DTSTART 

[2026-01-22 23:27:21.770] [ERROR] [calendar] Calendar Error. Could not fetch calendar:  https://calendar.google.com/calendar/ical/xxx.calendar.google.com/public/basic.ics iCal parsing failed: UNTIL rule part MUST have the same value type as DTSTART

khassel · 2026-01-22T22:49:54Z

Found the breaking entry:

DTSTART;VALUE=DATE:20160313
DTEND;VALUE=DATE:20160314
RRULE:FREQ=YEARLY;UNTIL=20190312;BYMONTHDAY=13;BYMONTH=3
DTSTAMP:20260122T223427Z
UID:a21aa3d4-c1c2-4d74-b929-535e168a566f
CREATED:20170127T230833Z
LAST-MODIFIED:20190330T190642Z
SEQUENCE:1
STATUS:CONFIRMED
SUMMARY:xxx
TRANSP:OPAQUE
X-MOZ-FAKED-MASTER:1
X-MOZ-GENERATION:2
END:VEVENT

I was able to solve the problem by deleting and recreating the entire calendar series in Google Calendar, but other users might encounter similar issues...

KristjanESPERANTO · 2026-01-22T22:52:50Z

Oh. I'll take a look at it by the weekend at the latest.

khassel · 2026-01-22T22:55:47Z

don't know if it is a real problem, that was the only entry with a RRULE in my birthday calendar and the UNTIL=20190312 was in the past ...

sdetweil · 2026-01-22T23:44:14Z

I think it is confused between full day and not for some reason, rrule until doesn’t use DATE: as it’s not required there

When parsing recurring events with VALUE=DATE (all-day events), preserve the VALUE=DATE parameter in the DTSTART string passed to rrule-temporal. This ensures that rrule-temporal can correctly validate that UNTIL values have the same type as DTSTART. Previously, date-only information was lost when building the RRULE string, causing rrule-temporal to incorrectly assume DTSTART was DATE-TIME and reject valid DATE-only UNTIL values with the error: "UNTIL rule part MUST have the same value type as DTSTART" The fix uses local date components (getFullYear, getMonth, getDate) which is consistent with how dateOnly dates are created in the dateParameter function using new Date(year, month, day). Fixes issue reported in MagicMirrorOrg/MagicMirror#4016 where Google Calendar birthday events with yearly RRULE and DATE-only UNTIL in the past failed to parse. Test coverage: - Yearly recurring DATE events with UNTIL - Monthly recurring DATE events with UNTIL - DATE events with COUNT instead of UNTIL - Edge case: VALUE=DATE with TZID (correctly ignores TZID)

KristjanESPERANTO · 2026-01-24T00:36:51Z

I think I found the root cause 🙂

The error comes from rrule-temporal v1.4.2, which added yesterday stricter RFC 5545 validation. The library now correctly enforces that UNTIL must match DTSTART's type - if DTSTART is a DATE (all-day event), UNTIL must also be a DATE.

The problem: node-ical never told rrule-temporal that DTSTART was VALUE=DATE, so the validation failed. The stricter validation is good - it just exposed a bug in node-ical that's been there all along.

Recreating the entry helped because Google Calendar probably removed the UNTIL or changed the format. But the underlying issue remains for other users with similar entries.

I've opened a fix: jens-maus/node-ical#432

This should handle Google Calendar birthdays and similar recurring all-day events correctly.

When parsing recurring events with VALUE=DATE (all-day events), preserve the VALUE=DATE parameter in the DTSTART string passed to rrule-temporal. This ensures that rrule-temporal can correctly validate that UNTIL values have the same type as DTSTART. Previously, date-only information was lost when building the RRULE string, causing rrule-temporal to incorrectly assume DTSTART was DATE-TIME and reject valid DATE-only UNTIL values with the error: "UNTIL rule part MUST have the same value type as DTSTART" The fix uses local date components (getFullYear, getMonth, getDate) which is consistent with how dateOnly dates are created in the dateParameter function using new Date(year, month, day). Fixes issue reported in MagicMirrorOrg/MagicMirror#4016 where Google Calendar birthday events with yearly RRULE and DATE-only UNTIL in the past failed to parse. Test coverage: - Yearly recurring DATE events with UNTIL - Monthly recurring DATE events with UNTIL - DATE events with COUNT instead of UNTIL - Edge case: VALUE=DATE with TZID (correctly ignores TZID)

This includes a new version of `node-ical` which should resolve a calendar issue that was reported [here](#4016 (comment)) and [here](#4010 (comment)).

Migrates the Newsfeed module to use the centralized HTTPFetcher class introduced in MagicMirrorOrg#4016, following the same pattern as the Calendar module. Changes: - Refactored NewsfeedFetcher from function constructor to ES6 class - Replaced manual fetch() + timer handling with HTTPFetcher composition - Uses structured error objects with translation keys - Inherits smart retry strategies (401/403, 429, 5xx backoff) - Inherits timeout handling (30s) and AbortController - Removed js/module_functions.js (scheduleTimer no longer needed) - Removed #module_functions import from package.json

This migrates the Newsfeed module to use the centralized HTTPFetcher class (introduced in #4016), following the same pattern as the Calendar module. This continues the refactoring effort to centralize HTTP error handling across all modules. ## Changes **NewsfeedFetcher:** - Refactored from function constructor to ES6 class (like the calendar module in #3959) - Replaced manual fetch() + timer handling with HTTPFetcher composition - Uses structured error objects with translation keys - Inherits smart retry strategies (401/403, 429, 5xx backoff) - Inherits timeout handling (30s) and AbortController **node_helper.js:** - Updated error handler to use `errorInfo.translationKey` - Simplified property access (`fetcher.url`, `fetcher.items`) **Cleanup:** - Removed `js/module_functions.js` (`scheduleTimer` no longer needed) - Removed `#module_functions` import from package.json ## Related Part of the HTTPFetcher migration effort started in #4016. Next candidate: Weather module (client-side → server-side migration).

… HTTPFetcher (#4032) This migrates the Weather module from client-side fetching to use the server-side centralized HTTPFetcher (introduced in #4016), following the same pattern as the Calendar and Newsfeed modules. ## Motivation This brings consistent error handling and better maintainability and completes the refactoring effort to centralize HTTP error handling across all default modules. Migrating to server-side providers with HTTPFetcher brings: - **Centralized error handling**: Inherits smart retry strategies (401/403, 429, 5xx backoff) and timeout handling (30s) - **Consistency**: Same architecture as Calendar and Newsfeed modules - **Security**: Possibility to hide API keys/secrets from client-side - **Performance**: Reduced API calls in multi-client setups - one server fetch instead of one per client - **Enabling possible future features**: e.g. server-side caching, rate limit monitoring, and data sharing with third-party modules ## Changes - All 10 weather providers now use HTTPFetcher for server-side fetching - Consistent error handling like Calendar and Newsfeed modules ## Breaking Changes None. Existing configurations continue to work. ## Testing To ensure proper functionality, I obtained API keys and credentials for all providers that require them. I configured all 10 providers in a carousel setup and tested each one individually. Screenshots for each provider are attached below demonstrating their working state. I even requested developer access from the Tempest/WeatherFlow team to properly test this provider. **Comprehensive test coverage**: A major advantage of the server-side architecture is the ability to thoroughly test providers with unit tests using real API response snapshots. Don't be alarmed by the many lines added in this PR - they are primarily test files and real-data mocks that ensure provider reliability. ## Review Notes I know this is an enormous change - I've been working on this for quite some time. Unfortunately, breaking it into smaller incremental PRs wasn't feasible due to the interdependencies between providers and the shared architecture. Given the scope, it's nearly impossible to manually review every change. To ensure quality, I've used both CodeRabbit and GitHub Copilot to review the code multiple times in my fork, and both provided extensive and valuable feedback. Most importantly, my test setup with all 10 providers working successfully is very encouraging. ## Related Part of the HTTPFetcher migration #4016. ## Screenshots <img width="1920" height="1080" alt="Ekrankopio de 2026-02-08 13-06-54" src="https://github.com/user-attachments/assets/2139f4d2-2a9b-4e49-8d0a-e4436983ed6e" /> <img width="1920" height="1080" alt="Ekrankopio de 2026-02-08 13-07-02" src="https://github.com/user-attachments/assets/880f7ce2-4e44-42d5-bfe4-5ce475cca7c2" /> <img width="1920" height="1080" alt="Ekrankopio de 2026-02-08 13-07-07" src="https://github.com/user-attachments/assets/abd89933-fe03-40ab-8a7c-41ae1ff99255" /> <img width="1920" height="1080" alt="Ekrankopio de 2026-02-08 13-07-12" src="https://github.com/user-attachments/assets/22225852-f0a9-4d33-87ab-0733ba30fad3" /> <img width="1920" height="1080" alt="Ekrankopio de 2026-02-08 13-07-17" src="https://github.com/user-attachments/assets/7a7192a5-f237-4060-85d7-6f50b9bef5af" /> <img width="1920" height="1080" alt="Ekrankopio de 2026-02-08 13-07-22" src="https://github.com/user-attachments/assets/df84d9f1-e531-4995-8da8-d6f2601b6a08" /> <img width="1920" height="1080" alt="Ekrankopio de 2026-02-08 13-07-27" src="https://github.com/user-attachments/assets/4cf391ac-db43-4b52-95f4-f5eadc5ea34d" /> <img width="1920" height="1080" alt="Ekrankopio de 2026-02-08 13-07-32" src="https://github.com/user-attachments/assets/8dd8e688-d47f-4815-87f6-7f2630f15d58" /> <img width="1920" height="1080" alt="Ekrankopio de 2026-02-08 13-07-37" src="https://github.com/user-attachments/assets/ee84a8bc-6b35-405a-b311-88658d9268dd" /> <img width="1920" height="1080" alt="Ekrankopio de 2026-02-08 13-07-42" src="https://github.com/user-attachments/assets/f941f341-453f-4d4d-a8d9-6b9158eb2681" /> Provider "Weather API" added later: <img width="1910" height="1080" alt="Ekrankopio de 2026-02-15 19-39-06" src="https://github.com/user-attachments/assets/3f0c8ba3-105c-4f90-8b2e-3a1be543d3d2" />

@angeldeejay

## Release Notes Thanks to: @angeldeejay, @in-voker, @JHWelch, @khassel, @KristjanESPERANTO, @rejas, @sdetweil > ⚠️ This release needs nodejs version >=22.21.1 <23 || >=24 (no change to previous release) [Compare to previous Release v2.34.0](v2.34.0...v2.25.0) > ⚠️ We introduced some internal changes with this release, please read [this forum post](https://forum.magicmirror.builders/topic/20138/upcoming-release-april-1-2026-breaking-changes-some-operational-changes) before upgrading! ### [core] - Prepare Release 2.35.0 (#4071) - docs: add security policy and vulnerability reporting guidelines (#4069) - refactor: simplify internal `require()` calls (#4056) - allow environment variables in cors urls (#4033) - fix cors proxy getting binary data (e.g. png, webp) (#4030) - fix: correct secret redaction and optimize loadConfig (#4031) - change loading config.js, allow variables in config.js and try to protect sensitive data (#4029) - remove kioskmode (#4027) - Add dark theme logo (#4026) - move custom.css from css to config (#4020) - move default modules from /modules/default to /defaultmodules (#4019) - update node versions in workflows (#4018) - [core] refactor: extract and centralize HTTP fetcher (#4016) - fix systeminformation not displaying electron version (#4012) - Update node-ical and support it's rrule-temporal changes (#4010) - Change default start scripts from X11 to Wayland (#4011) - refactor: unify favicon for index.html and Electron (#4006) - [core] run systeminformation in subprocess so the info is always displayed (#4002) - set next release dev number (#4000) ### [dependencies] - update dependencies (#4068) - update dependencies incl. electron to v41 (#4058) - chore: upgrade ESLint to v10 and fix newly surfaced issues (#4057) - chore: update ESLint and plugins, simplify config, apply new rules (#4052) - chore: update dependencies + add exports, files, and sideEffects fields to package.json (#4040) - [core] refactor: enable ESLint rule require-await and handle detected issues (#4038) - Update node-ical and other deps (#4025) - chore: update dependencies (#4021) - chore(eslint): migrate from eslint-plugin-vitest to @vitest/eslint-plugin and run rules only on test files (#4014) - Update deps as requested by dependabot (#4008) - update Collaboration.md and dependencies (#4001) ### [logging] - refactor: further logger clean-up (#4050) - Fix Node.js v25 logging prefix and modernize logger (#4049) ### [modules/calendar] - fix(calendar): make showEnd behavior more consistent across time formats (#4059) - test(calendar): fix hardcoded date in event shape test (#4055) - [calendar] refactor: delegate event expansion to node-ical's expandRecurringEvent (#4047) - calendar.js: remove useless hasCalendarURL function (#4028) - fix(calendar): update to node-ical 0.23.1 and fix full-day recurrence lookup (#4013) - fix(calendar): correct day-of-week for full-day recurring events across all timezones (#4004) ### [modules/newsfeed] - fix(newsfeed): fix full article view and add framing check (#4039) - [newsfeed] refactor: migrate to centralized HTTPFetcher (#4023) ### [modules/weather] - fix(weather): fix openmeteo forecast stuck in the past (#4064) - fix(weather): fix weathergov forecast day labels off by one (#4065) - weather: fixes for templates (#4054) - weather: add possibility to override njk's and css (#4051) - Use getDateString in openmeteo (#4046) - [weather] refactor: migrate to server-side providers with centralized HTTPFetcher (#4032) - [weather] feat: add Weather API Provider (#4036) ### [testing] - chore: remove obsolete Jest config and unit test global setup (#4044) - replace template_spec test with config_variables test (#4034) - refactor(clientonly): modernize code structure and add comprehensive tests (#4022) - Switch to undici Agent for HTTPS requests (#4015) - chore: migrate CI workflows to ubuntu-slim for faster startup times (#4007) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Kristjan ESPERANTO <35647502+KristjanESPERANTO@users.noreply.github.com> Co-authored-by: Bugsounet - Cédric <github@bugsounet.fr> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: sam detweiler <sdetweil@gmail.com> Co-authored-by: Veeck <github@veeck.de> Co-authored-by: veeck <gitkraken@veeck.de> Co-authored-by: Magnus <34011212+MagMar94@users.noreply.github.com> Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com> Co-authored-by: DevIncomin <56730075+Developer-Incoming@users.noreply.github.com> Co-authored-by: Nathan <n8nyoung@gmail.com> Co-authored-by: mixasgr <mixasgr@users.noreply.github.com> Co-authored-by: Savvas Adamtziloglou <savvas-gr@greeklug.gr> Co-authored-by: Konstantinos <geraki@gmail.com> Co-authored-by: OWL4C <124401812+OWL4C@users.noreply.github.com> Co-authored-by: BugHaver <43462320+bughaver@users.noreply.github.com> Co-authored-by: BugHaver <43462320+lsaadeh@users.noreply.github.com> Co-authored-by: Koen Konst <koenspero@gmail.com> Co-authored-by: Koen Konst <c.h.konst@avisi.nl> Co-authored-by: dathbe <github@beffa.us> Co-authored-by: Marcel <m-idler@users.noreply.github.com> Co-authored-by: Kevin G. <crazylegstoo@gmail.com> Co-authored-by: Jboucly <33218155+jboucly@users.noreply.github.com> Co-authored-by: Jboucly <contact@jboucly.fr> Co-authored-by: Jarno <54169345+jarnoml@users.noreply.github.com> Co-authored-by: Jordan Welch <JordanHWelch@gmail.com> Co-authored-by: Blackspirits <blackspirits@gmail.com> Co-authored-by: Samed Ozdemir <samed@xsor.io> Co-authored-by: in-voker <58696565+in-voker@users.noreply.github.com> Co-authored-by: Andrés Vanegas Jiménez <142350+angeldeejay@users.noreply.github.com>

khassel approved these changes Jan 20, 2026

View reviewed changes

rejas requested changes Jan 21, 2026

View reviewed changes

KristjanESPERANTO added 2 commits January 21, 2026 11:07

i18n: translate new messages

98ac345

fix: remove duplicates from pt translation

4c19d2c

KristjanESPERANTO requested a review from rejas January 21, 2026 10:13

khassel reviewed Jan 21, 2026

View reviewed changes

translations/pt.json Show resolved Hide resolved

khassel approved these changes Jan 21, 2026

View reviewed changes

khassel merged commit 34913bf into MagicMirrorOrg:develop Jan 22, 2026
9 checks passed

KristjanESPERANTO deleted the http-fetcher branch January 22, 2026 19:56

KristjanESPERANTO mentioned this pull request Jan 24, 2026

fix: preserve VALUE=DATE in DTSTART for RRULE parsing jens-maus/node-ical#432

Merged

KristjanESPERANTO mentioned this pull request Jan 27, 2026

chore: update dependencies #4021

Merged

khassel pushed a commit that referenced this pull request Jan 27, 2026

chore: update dependencies (#4021)

43503e8

This includes a new version of `node-ical` which should resolve a calendar issue that was reported [here](#4016 (comment)) and [here](#4010 (comment)).

KristjanESPERANTO mentioned this pull request Jan 28, 2026

[newsfeed] refactor: migrate to centralized HTTPFetcher #4023

Merged

This was referenced Feb 2, 2026

[weather] refactor: migrate to server-side providers with centralized HTTPFetcher KristjanESPERANTO/MagicMirror#35

Closed

[weather] refactor: migrate to server-side providers with centralized HTTPFetcher #4032

Merged

khassel mentioned this pull request Apr 1, 2026

Release 2.35.0 #4072

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[core] refactor: extract and centralize HTTP fetcher#4016

[core] refactor: extract and centralize HTTP fetcher#4016
khassel merged 3 commits intoMagicMirrorOrg:developfrom
KristjanESPERANTO:http-fetcher

KristjanESPERANTO commented Jan 20, 2026 •

edited

Loading

Uh oh!

rejas left a comment •

edited

Loading

Uh oh!

KristjanESPERANTO commented Jan 21, 2026

Uh oh!

Uh oh!

Uh oh!

khassel commented Jan 22, 2026

Uh oh!

khassel commented Jan 22, 2026

Uh oh!

KristjanESPERANTO commented Jan 22, 2026

Uh oh!

khassel commented Jan 22, 2026

Uh oh!

sdetweil commented Jan 22, 2026

Uh oh!

KristjanESPERANTO commented Jan 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

KristjanESPERANTO commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Architecture

Complexity Considerations

Future Benefits

Breaking Changes

Uh oh!

rejas left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KristjanESPERANTO commented Jan 21, 2026

Uh oh!

Uh oh!

Uh oh!

khassel commented Jan 22, 2026

Uh oh!

khassel commented Jan 22, 2026

Uh oh!

KristjanESPERANTO commented Jan 22, 2026

Uh oh!

khassel commented Jan 22, 2026

Uh oh!

sdetweil commented Jan 22, 2026

Uh oh!

KristjanESPERANTO commented Jan 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

KristjanESPERANTO commented Jan 20, 2026 •

edited

Loading

rejas left a comment •

edited

Loading